Skip to Content.
Sympa Menu

unclug - [unclug] trouble load balancing https with piranha (lvs, nanny, etc. on CentOS)

unclug AT lists.ibiblio.org

Subject: UNC Linux Users Group

List archive

Chronological Thread  
  • From: Palmer, Cristóbal <cmpalmer AT email.unc.edu>
  • To: "unclug AT lists.ibiblio.org" <unclug AT lists.ibiblio.org>
  • Subject: [unclug] trouble load balancing https with piranha (lvs, nanny, etc. on CentOS)
  • Date: Tue, 5 Aug 2014 15:24:36 +0000

For some context, I’m basically doing this (PDF link):

https://www.centos.org/docs/5/html/5.1/pdf/Virtual_Server_Administration.pdf

It has worked decently well for us (ibiblio) for over a decade with various
caveats (eg. only the round robin seems to work reliably). What we’ve never
done before (to my knowledge is load balance HTTPS, and I’m trying to change
that, but I’m hitting a wall.

Symptoms:

[root@lvs1 ~]# ipvsadm -L | head -n 3 && ipvsadm -L |grep guten
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP gutenberg.ibiblio.org:http rr
-> gutenweb3.lb.ibiblio.org:htt Masq 1 77 1316
-> gutenweb1.lb.ibiblio.org:htt Masq 1 75 1295
-> gutenweb2.lb.ibiblio.org:htt Masq 1 81 1297
TCP gutenberg.ibiblio.org:https rr

So what we see above is that for the http service (port 80) on the gutenberg
vip on the primary load balancer, we have three backend hosts that show up as
live and are handling connections nicely (~75 active, ~1300 inactive per
host). For https, however, we have no hosts. This is perplexing to me. We
should have one host (gutenweb3):

$ sudo cap guten_web invoke COMMAND="lsof -i :443"
triggering load callbacks
* 2014-08-05 11:16:04 executing `guten_web'
triggering start callbacks for `invoke'
* 2014-08-05 11:16:04 executing `multistage:ensure'
* 2014-08-05 11:16:04 executing `invoke'
* executing multiple commands in parallel
-> "else" :: "lsof -i :443"
-> "else" :: "lsof -i :443"
-> "else" :: "lsof -i :443"
servers: ["gutenweb1.int.ibiblio.org", "gutenweb2.int.ibiblio.org",
"gutenweb3.int.ibiblio.org"]
[gutenweb3.int.ibiblio.org] executing command
[gutenweb1.int.ibiblio.org] executing command
[gutenweb2.int.ibiblio.org] executing command
** [out :: gutenweb3.int.ibiblio.org] COMMAND PID USER FD TYPE DEVICE
SIZE/OFF NODE NAME
** [out :: gutenweb3.int.ibiblio.org] httpd 1731 root 6u IPv6 10671
0t0 TCP *:https (LISTEN)
** [out :: gutenweb3.int.ibiblio.org] httpd 6129 apache 6u IPv6 10671
0t0 TCP *:https (LISTEN)
** [out :: gutenweb3.int.ibiblio.org] httpd 6131 apache 6u IPv6 10671
0t0 TCP *:https (LISTEN)
** [out :: gutenweb3.int.ibiblio.org] httpd 6132 apache 6u IPv6 10671
0t0 TCP *:https (LISTEN)
** [out :: gutenweb3.int.ibiblio.org] httpd 6134 apache 6u IPv6 10671
0t0 TCP *:https (LISTEN)
** [out :: gutenweb3.int.ibiblio.org] httpd 6153 apache 6u IPv6 10671
0t0 TCP *:https (LISTEN)
** [out :: gutenweb3.int.ibiblio.org] httpd 6155 apache 6u IPv6 10671
0t0 TCP *:https (LISTEN)
** [out :: gutenweb3.int.ibiblio.org] httpd 6156 apache 6u IPv6 10671
0t0 TCP *:https (LISTEN)
** [out :: gutenweb3.int.ibiblio.org] httpd 6158 apache 6u IPv6 10671
0t0 TCP *:https (LISTEN)
** [out :: gutenweb3.int.ibiblio.org] httpd 6162 apache 6u IPv6 10671
0t0 TCP *:https (LISTEN)
** [out :: gutenweb3.int.ibiblio.org] httpd 6168 apache 6u IPv6 10671
0t0 TCP *:https (LISTEN)
** [out :: gutenweb3.int.ibiblio.org] httpd 6169 apache 6u IPv6 10671
0t0 TCP *:https (LISTEN)
** [out :: gutenweb3.int.ibiblio.org] httpd 6172 apache 6u IPv6 10671
0t0 TCP *:https (LISTEN)
** [out :: gutenweb3.int.ibiblio.org] httpd 6188 apache 6u IPv6 10671
0t0 TCP *:https (LISTEN)
** [out :: gutenweb3.int.ibiblio.org] httpd 6189 apache 6u IPv6 10671
0t0 TCP *:https (LISTEN)
command finished in 160ms
failed: "sh -c 'lsof -i :443'" on
gutenweb1.int.ibiblio.org,gutenweb2.int.ibiblio.org

So we have a host listening (because puppet told it to start listening), but
how are we checking that host?

[root@lvs1 ~]# ps -ef |grep nanny |grep 172.27.205.20
root 15621 15583 0 Jul30 ? 00:00:43 /usr/sbin/nanny -c -h
172.27.205.20 -p 80 -r 80 -s GET /check.php HTTP/1.0\r\n\r\n -x OK$ --regex
-a 15 -I /sbin/ipvsadm -t 10 -w 1 -V 152.19.134.47 -M m -U none --lvs
root 15630 15583 0 Jul30 ? 00:00:24 /usr/sbin/nanny -c -h
172.27.205.20 -p 443 -r 443 -e /usr/local/bin/check_https %h -x OK -a 15 -I
/sbin/ipvsadm -t 10 -w 1 -V 152.19.134.47 -M m -U none --lvs
[root@lvs1 ~]# cat /usr/local/bin/check_https
#!/bin/sh
## This file is managed by puppet. Do not modify except on the puppet master

IP=$1

## HTTP for testing
#wget --no-check-certificate --quiet -O /dev/stdout http://$IP/check.php
## HTTPS
#wget --no-check-certificate --quiet -O /dev/stdout https://$IP/check.php
RESULT=$(curl --insecure -s -S https://$IP/check.php 2>&1)

if [ "$RESULT" == "OK" ];then
echo -n OK
else
echo -n FAIL
## debug
#echo $RESULT
fi

So we have a service called “nanny” (aside: gendered and other problematic
terminology pops up all over the place in *nix work, but that is something I
will merely note but not address here given how long this post is already.)
and it is calling a silly little script. How is that script doing?

[root@lvs1 ~]# tail -n 5 /var/log/messages
Aug 5 11:04:14 lvs1 nanny[15629]: Trouble. Received results are not what we
expected from (172.27.205.28:443)
Aug 5 11:04:15 lvs1 nanny[15630]: Trouble. Received results are not what we
expected from (172.27.205.20:443)
Aug 5 11:04:24 lvs1 nanny[15628]: Trouble. Received results are not what we
expected from (172.27.205.27:443)
Aug 5 11:04:24 lvs1 nanny[15629]: Trouble. Received results are not what we
expected from (172.27.205.28:443)
Aug 5 11:04:25 lvs1 nanny[15630]: Trouble. Received results are not what we
expected from (172.27.205.20:443)

Looks like not well. But why?

[root@lvs1 ~]# /usr/local/bin/check_https 172.27.205.20
OK[root@lvs1 ~]# /usr/local/bin/check_https 172.27.205.28
FAIL[root@lvs1 ~]#

It looks like we ARE seeing the behavior we expect. So… *scratches head* …
what do y’all think?

Cheers,

Cristóbal Palmer
ibiblio.org



Archive powered by MHonArc 2.6.24.

Top of Page