Skip to Content.
Sympa Menu

internetworkers - RE: [internetworkers] Server Disappears Sporadically

internetworkers AT lists.ibiblio.org

Subject: Internetworkers: http://www.ibiblio.org/internetworkers/

List archive

Chronological Thread  
  • From: "Tony Spencer" <tony AT tonyspencer.com>
  • To: "'Internetworkers: http://www.ibiblio.org/internetworkers/'" <internetworkers AT lists.ibiblio.org>
  • Subject: RE: [internetworkers] Server Disappears Sporadically
  • Date: Thu, 2 Dec 2004 09:53:40 -0500

Thanks so much for the ideas Sil.

I think it may be that the firewall is to blame. I think it may have been
configured incorrectly. I installed the latest version, configured it, and
started testing ports and it seems to be working. I haven't had to reboot
since then and the firewall is no longer reporting 'offline' every 5
minutes.

The funky timestamps were really worrying me that the box had been hacked
and someone did a sloppy job of editing the logs. However, I have now
noticed this phenomenon across many logs so I think you must be correct
about the assumption that the system is updating its time from a timeserver.

Crossing fingers.

> -----Original Message-----
> From: Sil Greene [mailto:quack AT ibiblio.org]
> Sent: Wednesday, December 01, 2004 5:19 PM
> To: Internetworkers: http://www.ibiblio.org/internetworkers/
> Subject: RE: [internetworkers] Server Disappears Sporadically
>
>
> Tony,
>
> I'm not a firewall expert, and I haven't worked with APF
> before. With
> that said, I'm not seeing the APF logs that indicate it's
> opening inbound
> ports for HTTP, FTP, and other traffic. It does seem to be
> opening SSH
> (port 22). I suspect all traffic should be allowed when APF
> indicates
> it's offline.
>
> The "default (ingress) input drop" message suggests that, at
> that point in
> time, everything but SSH and ping (and a telnet straight to port 22)
> should fail. I'm not sure what the significance is of the
> skip-backwards-1-hour phenomenon (notice this is visible in
> both Apache
> and APF logs) -- perhaps the server is updating the system
> time from a
> timeserver? Is this seen in your system logs as well? Any
> clues from an
> NTP daemon in any logs?
>
> How often do you have to reboot the system?
>
> --Sil
>
>
>
> from Tony Spencer:
> .:
> .:Yes. Much thanks in advance for taking a look.
> .:
> .:Dec 01 13:00:00 server1 apf(10221): flushing & zeroing
> chain policies
> .:Dec 01 13:00:00 server1 apf(10221): firewall offline
> .:Dec 01 13:05:00 server1 apf(10515): flushing & zeroing
> chain policies
> .:Dec 01 13:05:00 server1 apf(10515): firewall offline
> .:Dec 01 13:10:00 server1 apf(10741): flushing & zeroing
> chain policies
> .:Dec 01 13:10:00 server1 apf(10741): firewall offline
> .:Dec 01 13:15:01 server1 apf(10991): flushing & zeroing
> chain policies
> .:Dec 01 13:15:01 server1 apf(10991): firewall offline
> .:Dec 01 13:20:00 server1 apf(11281): flushing & zeroing
> chain policies
> .:Dec 01 13:20:00 server1 apf(11281): firewall offline
> .:Dec 01 13:25:00 server1 apf(11516): flushing & zeroing
> chain policies
> .:Dec 01 13:25:00 server1 apf(11516): firewall offline
> .:Dec 01 13:30:00 server1 apf(11805): flushing & zeroing
> chain policies
> .:Dec 01 13:30:00 server1 apf(11805): firewall offline
> .:Dec 01 14:38:24 server1 apf(1868): activating firewall
> .:Dec 01 14:38:25 server1 apf(1904): determined eth0 has
> address xx.xx.xx.xx
> .:Dec 01 14:38:25 server1 apf(1904): loading sysctl.rules
> .:Dec 01 14:38:25 server1 apf(1904): loading preroute.rules
> .:Dec 01 14:38:26 server1 apf(1904): loading prelog.rules
> .:Dec 01 14:38:26 server1 apf(1904): loading bt.rules
> .:Dec 01 14:38:26 server1 apf(1904): loading 192.168.234.235.rules
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound tcp port 22 on
> .:192.168.234.235
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 3 on
> .:192.168.234.235
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 5 on
> .:192.168.234.235
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 11 on
> .:192.168.234.235
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 0 on
> .:192.168.234.235
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 30 on
> .:192.168.234.235
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 8 on
> .:192.168.234.235
> .:Dec 01 14:38:26 server1 apf(1904): loading main.rules
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound tcp port 22 on
> .:xx.xx.xx.xx
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 3 on
> .:xx.xx.xx.xx
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 5 on
> .:xx.xx.xx.xx
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 11 on
> .:xx.xx.xx.xx
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 0 on
> .:xx.xx.xx.xx
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 30 on
> .:xx.xx.xx.xx
> .:Dec 01 14:38:26 server1 apf(1904): opening inbound icmp type 8 on
> .:xx.xx.xx.xx
> .:Dec 01 14:38:26 server1 apf(1904): default (egress) output accept
> .:Dec 01 14:38:26 server1 apf(1904): default (ingress) input drop
> .:Dec 01 14:38:27 server1 apf(1868): firewall initalized
> .:Dec 01 14:38:51 server1 apf(2957): activating firewall
> .:Dec 01 13:35:21 server1 apf(2998): determined eth0 has
> address xx.xx.xx.xx
> .:Dec 01 13:35:21 server1 apf(2998): loading sysctl.rules
> .:Dec 01 13:35:21 server1 apf(2998): loading preroute.rules
> .:Dec 01 13:35:21 server1 apf(2998): loading prelog.rules
> .:Dec 01 13:35:21 server1 apf(2998): loading bt.rules
> .:Dec 01 13:35:21 server1 apf(2998): loading 192.168.234.235.rules
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound tcp port 22 on
> .:192.168.234.235
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 3 on
> .:192.168.234.235
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 5 on
> .:192.168.234.235
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 11 on
> .:192.168.234.235
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 0 on
> .:192.168.234.235
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 30 on
> .:192.168.234.235
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 8 on
> .:192.168.234.235
> .:Dec 01 13:35:21 server1 apf(2998): loading main.rules
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound tcp port 22 on
> .:xx.xx.xx.xx
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 3 on
> .:xx.xx.xx.xx
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 5 on
> .:xx.xx.xx.xx
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 11 on
> .:xx.xx.xx.xx
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 0 on
> .:xx.xx.xx.xx
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 30 on
> .:xx.xx.xx.xx
> .:Dec 01 13:35:21 server1 apf(2998): opening inbound icmp type 8 on
> .:xx.xx.xx.xx
> .:Dec 01 13:35:21 server1 apf(2998): default (egress) output accept
> .:Dec 01 13:35:21 server1 apf(2998): default (ingress) input drop
> .:Dec 01 13:35:22 server1 apf(2957): firewall initalized
> .:Dec 01 13:40:00 server1 apf(3582): flushing & zeroing chain policies
> .:Dec 01 13:40:00 server1 apf(3582): firewall offline
> .:Dec 01 13:45:00 server1 apf(3969): flushing & zeroing chain policies
> .:Dec 01 13:45:00 server1 apf(3969): firewall offline
> .:
> .:> -----Original Message-----
> .:> From: Sil Greene [mailto:quack AT ibiblio.org]
> .:> Sent: Wednesday, December 01, 2004 3:12 PM
> .:> To: Internetworkers: http://www.ibiblio.org/internetworkers/
> .:> Subject: Re: [internetworkers] Server Disappears Sporadically
> .:>
> .:>
> .:> Tony, do you also have the firewall logs from that time period?
> .:> --s
> .:>
> .:>
> .:>
> .:>
> .:> from Tony Spencer:
> .:> .:
> .:> .:I lease a server running RedHat Linux. Recently it has
> .:> started disappearing .:for no apparent reason. When I say
> .:> disappearing I mean that there is no .:response on port 80, I
> .:> can't FTP to it, I can't SSH2, and I can't even ping .:it. I
> .:> have to request my ISP to reboot it. It always comes back up
> .:> with no .:problem after a reboot. I'm not a sysadmin and
> .:> I've usually managed fine .:but this one is boggling me. Can
> .:> anyone give me any more pointers for what .:to look for? I'm
> .:> wondering if the firewall is closing everything for some .:reason.
> .:> .:
> .:> .:Here is the situation:
> .:> .:RedHat v3.1.0
> .:> .:2 GB of RAM
> .:> .:Running two websites. Load rarely tops 0.2 and usually is
> .:> 0 to 0.1 I ran .:chkrootkit as well as Rootkit Hunter and
> .:> found no problems.
> .:> .:Telnet is off.
> .:> .:Runs APF firewall with the following ports open:
> .:> .:TCP_CPORTS="
> .:>
> .:21,22,25,26,53,80,110,143,443,2082,2083,2086,2087,2095,2096,19638"
> .:> .:UDP_CPORTS="53"
> .:> .:
> .:> .:Here are logs just before and after the disappearing act.
> .:> I think the .:proftpd stuff is just a monitor on the box
> .:> checking whether or not FTP is .:working. I don't understand
> .:> why the apache logs actually go backwards in .:time at one
> .:> point (14:38:37 to 13:35:21).
> .:> .:
> .:> .:-----------------------------------------/var/log/messages--
> .:> ----------------
> .:> .:----------------
> .:> .:Dec 1 11:48:47 server1 proftpd[3892]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed.
> .:> .:Dec 1 11:57:07 server1 proftpd[4500]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
> .:> disconnected Dec 1
> .:> .:11:57:07 server1 proftpd[4500]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed.
> .:> .:Dec 1 12:05:28 server1 proftpd[5466]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
> .:> disconnected Dec 1
> .:> .:12:05:28 server1 proftpd[5466]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed.
> .:> .:Dec 1 12:13:48 server1 proftpd[6336]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
> .:> disconnected Dec 1
> .:> .:12:13:48 server1 proftpd[6336]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed.
> .:> .:Dec 1 12:22:08 server1 proftpd[7163]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
> .:> disconnected Dec 1
> .:> .:12:22:08 server1 proftpd[7163]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed.
> .:> .:Dec 1 12:30:28 server1 proftpd[8289]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
> .:> disconnected Dec 1
> .:> .:12:30:28 server1 proftpd[8289]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed.
> .:> .:Dec 1 12:38:48 server1 proftpd[8787]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
> .:> disconnected Dec 1
> .:> .:12:38:48 server1 proftpd[8787]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed.
> .:> .:Dec 1 12:47:08 server1 proftpd[9280]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
> .:> disconnected Dec 1
> .:> .:12:47:09 server1 proftpd[9280]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed.
> .:> .:Dec 1 12:55:29 server1 proftpd[9713]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
> .:> disconnected Dec 1
> .:> .:12:55:29 server1 proftpd[9713]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed.
> .:> .:Dec 1 13:03:49 server1 proftpd[10167]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
> .:> disconnected Dec 1
> .:> .:13:03:49 server1 proftpd[10167]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed.
> .:> .:Dec 1 13:12:09 server1 proftpd[10635]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
> .:> disconnected Dec 1
> .:> .:13:12:09 server1 proftpd[10635]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed.
> .:> .:Dec 1 13:15:29 server1 proftpd[11067]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session opened.
> .:> .:Dec 1 13:15:29 server1 proftpd[11067]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed.
> .:> .:Dec 1 13:28:49 server1 proftpd[11484]: server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP no transfer timeout,
> .:> disconnected Dec 1 .:13:28:50 server1 proftpd[11484]:
> .:> server1.mydomain.com
> .:> .:(localhost[127.0.0.1]) - FTP session closed.
> .:> .:Dec 1 14:38:13 server1 syslogd 1.4.1: restart.
> .:> .:
> .:> .:--------------------------------Apache
> .:> error_log---------------------------
> .:> .:[Wed Dec 1 13:20:14 2004] [error] [client 65.33.122.230]
> .:> File does not
> .:> .:exist: /home/myuser/public_html/images/trans.gif
> .:> .:[Wed Dec 1 13:20:18 2004] [error] [client 66.243.50.46]
> .:> File does not
> .:> .:exist: /home/myuser/public_html/images/trans.gif
> .:> .:[Wed Dec 1 13:23:02 2004] [error] [client 66.206.96.196]
> .:> File does not
> .:> .:exist: /home/myuser/public_html/images/trans.gif
> .:> .:[Wed Dec 1 13:24:24 2004] [error] [client 209.30.109.245]
> .:> File does not
> .:> .:exist: /home/myuser/public_html/images/trans.gif
> .:> .:[Wed Dec 1 13:25:04 2004] [error] [client 24.119.238.68]
> .:> File does not
> .:> .:exist: /home/myuser/public_html/images/trans.gif
> .:> .:[Wed Dec 1 13:26:26 2004] [error] [client 66.139.27.254]
> .:> File does not
> .:> .:exist: /home/myuser/public_html/images/trans.gif
> .:> .:[Wed Dec 1 14:38:37 2004] [warn] pid file
> .:> /usr/local/apache/logs/httpd.pid .:overwritten -- Unclean
> .:> shutdown of previous Apache run?
> .:> .:[Wed Dec 1 14:38:37 2004] [crit] (98)Address already in
> .:> use: make_sock:
> .:> .:could not bind to port 443
> .:> .:[Wed Dec 1 14:38:37 2004] [notice] Apache/1.3.29 (Unix)
> .:> Resin/2.1.12
> .:> .:mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 P
> .:> .:HP/4.3.3 FrontPage/5.0.2.2634 mod_ssl/2.8.16 OpenSSL/0.9.7a
> .:> configured -- .:resuming normal operations [Wed Dec 1
> .:> 14:38:37 2004] [notice] suEXEC .:mechanism enabled (wrapper:
> .:> .:/usr/local/apache/bin/suexec)
> .:> .:[Wed Dec 1 14:38:37 2004] [notice] Accept mutex: sysvsem
> .:> (Default: sysvsem) .:[Wed Dec 1 13:35:21 2004] [notice]
> .:> SIGUSR1 received. Doing graceful .:restart [Wed Dec 1
> .:> 13:35:21 2004] [notice] Apache/1.3.29 (Unix)
> .:> .:Resin/2.1.12
> .:> .:mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 P
> .:> .:HP/4.3.3 FrontPage/5.0.2.2634 mod_ssl/2.8.16 OpenSSL/0.9.7a
> .:> configured -- .:resuming normal operations [Wed Dec 1
> .:> 13:35:21 2004] [notice] suEXEC .:mechanism enabled (wrapper:
> .:> .:/usr/local/apache/bin/suexec)
> .:> .:[Wed Dec 1 13:35:21 2004] [notice] Accept mutex: sysvsem
> .:> (Default: sysvsem) .:[Wed Dec 1 13:35:21 2004] [error]
> .:> [client 12.219.125.81] File does not
> .:> .:exist: /home/myuser/public_html/free-stuff/free-download-ant
> .:> .:
> .:> .:
> .:> .:
> .:> .:---
> .:> .:Come and play at the InterNetWorkers Web site!
> .:> http://www.ibiblio.org/internetworkers/
> .:> .:You are currently subscribed to InterNetWorkers mailing
> .:> list .:To unsubscribe visit
> .:> http://lists.ibiblio.org/mailman/listinfo/internetworkers
> .:> .:
> .:>
> .:> --
> .:> "If you put your supper dish to your ears you can hear the
> .:> sounds of a restaurant." --Snoopy
> .:> ---
> .:> Come and play at the InterNetWorkers Web site!
> .:> http://www.ibiblio.org/internetworkers/
> .:> You are currently subscribed to InterNetWorkers mailing list
> .:> To unsubscribe visit
> .:> http://lists.ibiblio.org/mailman/listinfo/internetworkers
> .:>
> .:>
> .:
> .:---
> .:Come and play at the InterNetWorkers Web site!
> http://www.ibiblio.org/internetworkers/
> .:You are currently subscribed to InterNetWorkers mailing list
> .:To unsubscribe visit
> http://lists.ibiblio.org/mailman/listinfo/internetworkers
> .:
>
> --
> "If you put your supper dish to your ears you can hear the sounds
> of a restaurant." --Snoopy
> ---
> Come and play at the InterNetWorkers Web site!
> http://www.ibiblio.org/internetworkers/
> You are currently subscribed to InterNetWorkers mailing list
> To unsubscribe visit
> http://lists.ibiblio.org/mailman/listinfo/internetworkers
>
>





Archive powered by MHonArc 2.6.24.

Top of Page