UNC Linux Users Group

[Palmer, Cristóbal <cmpalmer AT email.unc.edu>]

For some context, I’m basically doing this (PDF link):

https://www.centos.org/docs/5/html/5.1/pdf/Virtual_Server_Administration.pdf

It has worked decently well for us (ibiblio) for over a decade with various 
caveats (eg. only the round robin seems to work reliably). What we’ve never 
done before (to my knowledge is load balance HTTPS, and I’m trying to change 
that, but I’m hitting a wall.

Symptoms:

[root@lvs1 ~]# ipvsadm -L | head -n 3 && ipvsadm -L |grep guten
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  gutenberg.ibiblio.org:http rr
  -> gutenweb3.lb.ibiblio.org:htt Masq    1      77         1316      
  -> gutenweb1.lb.ibiblio.org:htt Masq    1      75         1295      
  -> gutenweb2.lb.ibiblio.org:htt Masq    1      81         1297      
TCP  gutenberg.ibiblio.org:https rr

So what we see above is that for the http service (port 80) on the gutenberg 
vip on the primary load balancer, we have three backend hosts that show up as 
live and are handling connections nicely (~75 active, ~1300 inactive per 
host). For https, however, we have no hosts. This is perplexing to me. We 
should have one host (gutenweb3):

$ sudo cap guten_web invoke COMMAND="lsof -i :443"
    triggering load callbacks
  * 2014-08-05 11:16:04 executing `guten_web'
    triggering start callbacks for `invoke'
  * 2014-08-05 11:16:04 executing `multistage:ensure'
  * 2014-08-05 11:16:04 executing `invoke'
  * executing multiple commands in parallel
    -> "else" :: "lsof -i :443"
    -> "else" :: "lsof -i :443"
    -> "else" :: "lsof -i :443"
    servers: ["gutenweb1.int.ibiblio.org", "gutenweb2.int.ibiblio.org", 
"gutenweb3.int.ibiblio.org"]
    [gutenweb3.int.ibiblio.org] executing command
    [gutenweb1.int.ibiblio.org] executing command
    [gutenweb2.int.ibiblio.org] executing command
 ** [out :: gutenweb3.int.ibiblio.org] COMMAND  PID   USER   FD   TYPE DEVICE 
SIZE/OFF NODE NAME
 ** [out :: gutenweb3.int.ibiblio.org] httpd   1731   root    6u  IPv6  10671 
     0t0  TCP *:https (LISTEN)
 ** [out :: gutenweb3.int.ibiblio.org] httpd   6129 apache    6u  IPv6  10671 
     0t0  TCP *:https (LISTEN)
 ** [out :: gutenweb3.int.ibiblio.org] httpd   6131 apache    6u  IPv6  10671 
     0t0  TCP *:https (LISTEN)
 ** [out :: gutenweb3.int.ibiblio.org] httpd   6132 apache    6u  IPv6  10671 
     0t0  TCP *:https (LISTEN)
 ** [out :: gutenweb3.int.ibiblio.org] httpd   6134 apache    6u  IPv6  10671 
     0t0  TCP *:https (LISTEN)
 ** [out :: gutenweb3.int.ibiblio.org] httpd   6153 apache    6u  IPv6  10671 
     0t0  TCP *:https (LISTEN)
 ** [out :: gutenweb3.int.ibiblio.org] httpd   6155 apache    6u  IPv6  10671 
     0t0  TCP *:https (LISTEN)
 ** [out :: gutenweb3.int.ibiblio.org] httpd   6156 apache    6u  IPv6  10671 
     0t0  TCP *:https (LISTEN)
 ** [out :: gutenweb3.int.ibiblio.org] httpd   6158 apache    6u  IPv6  10671 
     0t0  TCP *:https (LISTEN)
 ** [out :: gutenweb3.int.ibiblio.org] httpd   6162 apache    6u  IPv6  10671 
     0t0  TCP *:https (LISTEN)
 ** [out :: gutenweb3.int.ibiblio.org] httpd   6168 apache    6u  IPv6  10671 
     0t0  TCP *:https (LISTEN)
 ** [out :: gutenweb3.int.ibiblio.org] httpd   6169 apache    6u  IPv6  10671 
     0t0  TCP *:https (LISTEN)
 ** [out :: gutenweb3.int.ibiblio.org] httpd   6172 apache    6u  IPv6  10671 
     0t0  TCP *:https (LISTEN)
 ** [out :: gutenweb3.int.ibiblio.org] httpd   6188 apache    6u  IPv6  10671 
     0t0  TCP *:https (LISTEN)
 ** [out :: gutenweb3.int.ibiblio.org] httpd   6189 apache    6u  IPv6  10671 
     0t0  TCP *:https (LISTEN)
    command finished in 160ms
failed: "sh -c 'lsof -i :443'" on 
gutenweb1.int.ibiblio.org,gutenweb2.int.ibiblio.org

So we have a host listening (because puppet told it to start listening), but 
how are we checking that host?

[root@lvs1 ~]# ps -ef |grep nanny |grep 172.27.205.20
root     15621 15583  0 Jul30 ?        00:00:43 /usr/sbin/nanny -c -h 
172.27.205.20 -p 80 -r 80 -s GET /check.php HTTP/1.0\r\n\r\n -x OK$ --regex 
-a 15 -I /sbin/ipvsadm -t 10 -w 1 -V 152.19.134.47 -M m -U none --lvs
root     15630 15583  0 Jul30 ?        00:00:24 /usr/sbin/nanny -c -h 
172.27.205.20 -p 443 -r 443 -e /usr/local/bin/check_https %h -x OK -a 15 -I 
/sbin/ipvsadm -t 10 -w 1 -V 152.19.134.47 -M m -U none --lvs
[root@lvs1 ~]# cat /usr/local/bin/check_https
#!/bin/sh
## This file is managed by puppet. Do not modify except on the puppet master

IP=$1

## HTTP for testing
#wget --no-check-certificate --quiet -O /dev/stdout http://$IP/check.php
## HTTPS
#wget --no-check-certificate --quiet -O /dev/stdout https://$IP/check.php
RESULT=$(curl --insecure -s -S https://$IP/check.php 2>&1)

if [ "$RESULT" == "OK" ];then
  echo -n OK
else
  echo -n FAIL
  ## debug
  #echo $RESULT
fi

So we have a service called “nanny” (aside: gendered and other problematic 
terminology pops up all over the place in *nix work, but that is something I 
will merely note but not address here given how long this post is already.) 
and it is calling a silly little script. How is that script doing?

[root@lvs1 ~]# tail -n 5 /var/log/messages
Aug  5 11:04:14 lvs1 nanny[15629]: Trouble. Received results are not what we 
expected from (172.27.205.28:443)
Aug  5 11:04:15 lvs1 nanny[15630]: Trouble. Received results are not what we 
expected from (172.27.205.20:443)
Aug  5 11:04:24 lvs1 nanny[15628]: Trouble. Received results are not what we 
expected from (172.27.205.27:443)
Aug  5 11:04:24 lvs1 nanny[15629]: Trouble. Received results are not what we 
expected from (172.27.205.28:443)
Aug  5 11:04:25 lvs1 nanny[15630]: Trouble. Received results are not what we 
expected from (172.27.205.20:443)

Looks like not well. But why?

[root@lvs1 ~]# /usr/local/bin/check_https 172.27.205.20
OK[root@lvs1 ~]# /usr/local/bin/check_https 172.27.205.28
FAIL[root@lvs1 ~]# 

It looks like we ARE seeing the behavior we expect. So… *scratches head* … 
what do y’all think?

Cheers,
—
Cristóbal Palmer
ibiblio.org