Skip to Content.
Sympa Menu

unclug - Re: [unclug] trouble load balancing https with piranha (lvs, nanny, etc. on CentOS)

unclug AT lists.ibiblio.org

Subject: UNC Linux Users Group

List archive

Chronological Thread  
  • From: Palmer, Cristóbal <cmpalmer AT email.unc.edu>
  • To: "unclug AT lists.ibiblio.org" <unclug AT lists.ibiblio.org>
  • Subject: Re: [unclug] trouble load balancing https with piranha (lvs, nanny, etc. on CentOS)
  • Date: Fri, 8 Aug 2014 14:18:56 +0000


On Aug 5, 2014, at 11:24 AM, Palmer, Cristóbal <cmpalmer AT email.unc.edu> wrote:

> [root@lvs1 ~]# ps -ef |grep nanny |grep 172.27.205.20


I made a change and now I see:

[root@lvs1 ~]# ps -ef |grep nanny |grep -i https
root 11055 11025 0 09:39 ? 00:00:00 /usr/sbin/nanny -c -h
172.27.205.27 -p 443 -r 443 -e /usr/local/bin/check_https %h -x OK --regex -a
15 -I /sbin/ipvsadm -t 10 -w 1 -V 152.19.134.47 -M m -U none --lvs
root 11056 11025 0 09:39 ? 00:00:00 /usr/sbin/nanny -c -h
172.27.205.28 -p 443 -r 443 -e /usr/local/bin/check_https %h -x OK --regex -a
15 -I /sbin/ipvsadm -t 10 -w 1 -V 152.19.134.47 -M m -U none --lvs
root 11057 11025 0 09:39 ? 00:00:00 /usr/sbin/nanny -c -h
172.27.205.20 -p 443 -r 443 -e /usr/local/bin/check_https %h -x OK --regex -a
15 -I /sbin/ipvsadm -t 10 -w 1 -V 152.19.134.47 -M m -U none —lvs

The full config block:

——8< snip 8<——

$ git diff !$
git diff /etc/puppet/modules/lvs/files/lvs.cf.production
diff --git a/modules/lvs/files/lvs.cf.production
b/modules/lvs/files/lvs.cf.production
index 225a0d9..42c5257 100755
--- a/modules/lvs/files/lvs.cf.production
+++ b/modules/lvs/files/lvs.cf.production
@@ -223,6 +223,7 @@ virtual etree.ibiblio.org-http {

#------------------------------------------------------------#
# gutenberg.ibiblio.org => eth0:5
+# HTTP
virtual gutenberg.ibiblio.org-http {
active = 1
address = 152.19.134.47 eth0:5
@@ -258,6 +259,43 @@ virtual gutenberg.ibiblio.org-http {
}

#------------------------------------------------------------#
+# gutenberg.ibiblio.org => eth0:5
+# HTTPS
+virtual gutenberg.ibiblio.org-https {
+ active = 1
+ address = 152.19.134.47 eth0:5
+ vip_nmask = 255.255.255.192
+ port = 443
+ send_program = "/usr/local/bin/check_https %h"
+ expect = "OK"
+ use_regex = 1
+ load_monitor = none
+ scheduler = rr
+ protocol = tcp
+ timeout = 10
+ reentry = 15
+ quiesce_server = 0
+ server gutenweb1.lb-https {
+ address = 172.27.205.27
+ active = 1
+ port = 443
+ weight = 1
+ }
+ server gutenweb2.lb-https {
+ address = 172.27.205.28
+ active = 1
+ port = 443
+ weight = 1
+ }
+ server gutenweb3.lb-https {
+ address = 172.27.205.20
+ active = 1
+ port = 443
+ weight = 1
+ }
+}
+
+#------------------------------------------------------------#
# distro.ibiblio.org => eth0:6
virtual distro.ibiblio.org-http {
active = 1

——8< snip 8<——

… but still I see failures:

[root@lvs1 ~]# tail -n 3 /var/log/messages


Aug 8 09:52:54 lvs1 nanny[11056]: Trouble. Received results are not what we
expected from (172.27.205.28:443)
Aug 8 09:52:54 lvs1 nanny[11055]: Trouble. Received results are not what we
expected from (172.27.205.27:443)
Aug 8 09:52:59 lvs1 nanny[11057]: Trouble. Received results are not what we
expected from (172.27.205.20:443)


A clue, but with a less-than-helpful error:

[root@lvs1 ~]# /usr/sbin/nanny -c -h 172.27.205.20 -p 443 -r 443 -e
/usr/local/bin/check_https %h -x OK --regex -I /sbin/ipvsadm -w 1 -V
152.19.134.47 --norun
nanny: unexpected arguments


Yes…. Quite. Well what are the arguments?

[root@lvs1 ~]# /usr/sbin/nanny --usage
Usage: nanny [-ncTquv?] [-t|--interval INT] [-I|--ipvsadm STRING]
[-U|--loadcmd STRING] [-M|--method STRING] [-n|--nodaemon] [-c|--nofork]
[-p|--port INT] [-r|--rport INT] [-a|--reentrytime INT]
[-S|--scale INT] [-h|--server STRING] [-e|--send_program STRING]
[-s|--send_string STRING] [-x|--expect_string STRING] [--regex]
[-T|--suppress] [-q|--quiesce_server] [-u|--udp]
[-v|--verbose] [-V|--virtaddress STRING] [-f|--fwmark INT]
[-w|--weight INT] [--test-start] [--norun] [--version] [--fos] [--lvs]
[-?|--help] [—usage]


Possibly it is reading "-e /usr/local/bin/check_https %h” as multiple
arguments? Let’s try quoting:

[root@lvs1 ~]# /usr/sbin/nanny -c -h 172.27.205.20 -p 443 -r 443 -e
'/usr/local/bin/check_https %h' -x OK --regex -I /sbin/ipvsadm -w 1 -V
152.19.134.47 --norun
nanny: unknown service type

???

So if I actually go with the single quotes in the lvs.cf I get this in the
logs:

Aug 8 10:00:07 lvs2 nanny[1067]: The following exited abnormally:
Aug 8 10:00:07 lvs2 nanny[1067]: Ran the external sending program to
(172.27.205.27:443) but didn't get anything back
Aug 8 10:00:07 lvs2 nanny[1069]: The following exited abnormally:
Aug 8 10:00:07 lvs2 nanny[1069]: Ran the external sending program to
(172.27.205.20:443) but didn't get anything back
Aug 8 10:00:07 lvs2 nanny[1068]: The following exited abnormally:
Aug 8 10:00:07 lvs2 nanny[1068]: Ran the external sending program to
(172.27.205.28:443) but didn't get anything back

but the hosts show up as available:

[root@lvs2 ~]# ipvsadm -L |tail -n 4
TCP gutenberg.ibiblio.org:https rr
-> gutenweb3.lb.ibiblio.org:htt Masq 1 0 0
-> gutenweb1.lb.ibiblio.org:htt Masq 1 0 0
-> gutenweb2.lb.ibiblio.org:htt Masq 1 0 0


I would love some feedback on what might be going on here. Anybody want to
meet up and look at this with me? I keep digging around and I find forum
posts like this one:

https://www.centos.org/forums/viewtopic.php?t=22135

that indicate that I’m not alone, but I can’t tell where in this chain we
might be getting caught up. If only there were a stronger validator for the
lvs.cf and/or better error messages….

Cheers,

Cristóbal Palmer





Archive powered by MHonArc 2.6.24.

Top of Page