When doing server upgrades with multiple servers, the ideal way is to:
1. take one instance out of the pool 2. drain connections on it 3. upgrade it 4. put it back into the pool 5. back to #1.
The various techniques can be categorized as:
1. application-level 2. load-balancer-level 3. OS-level
The most graceful method is to use an application-level feature, since the application knows what its worker status is.
For example, with httpd on CentOS or Redhat, either use the apachectl command, or add the graceful-stop option to /etc/init.d/httpd:
#!/bin/bash set -e echo "info: Draining connections ..." apachectl graceful-stop echo "info: You have 5 minutes to start and finish your upgrade." sleep 300 apachectl start echo "info: httpd restarted!" exit 0
If we didn’t have an application-specific way to do that, we could use iptables:
#!/bin/bash set -e iptables -I INPUT -j DROP -p tcp --syn --destination-port 80 echo "info: Draining connections ..." sleep 60 echo "info: You have 5 minutes to start and finish your upgrade" sleep 300 iptables -D INPUT -j DROP -p tcp --syn --destination-port 80 echo "info: iptables allowing new incoming connections!" exit 0
With HAProxy we can do this on the HAProxy host (do yum -y install socat first):
#!/bin/bash set -e echo "set server application-backend/www0 state drain" | socat unix-connect:/var/run/haproxy.sock stdio echo "info: Draining connections ..." sleep 60 echo "set server application-backend/www0 state maint" | socat unix-connect:/var/run/haproxy.sock stdio echo "info: You have 5 minutes to start and finish your upgrade" sleep 300 echo "set server application-backend/www0 state ready" | socat unix-connect:/var/run/haproxy.sock stdio echo "info: haproxy allowing new incoming connections!" exit 0
Sample HAProxy “show stat” output while www0 is draining (notice the “DRAIN” status):
[root@gw ~]# echo "show info" | socat unix-connect:/var/run/haproxy.sock stdio Name: HAProxy Version: 1.5.10 [..] [root@gw ~]# echo "show stat" | socat unix-connect:/var/run/haproxy.sock stdio # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime, http-in,FRONTEND,,,0,1,2000,6,998,19219,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,1,,,,0,5,1,0,0,0,,0,1,6,,,0,0,0,0,,,,,,,, https-in,FRONTEND,,,0,20,2000,12,294,2556,0,0,11,,,,,OPEN,,,,,,,,,1,3,0,,,,0,0,0,2,,,,0,0,1,11,0,0,,0,2,12,,,0,0,0,0,,,,,,,, application-backend,www0,0,0,0,1,5000,3,583,1078,,0,,0,0,0,0,DRAIN,1,1,0,0,0,385,0,,1,4,1,,3,,2,0,,1,L7OK,301,0,0,2,1,0,0,0,0,,,,0,0,,,,,442,Moved Permanently,,0,0,0,1, application-backend,www1,0,0,0,1,5000,4,709,18640,,0,,0,0,0,0,UP,1,1,0,0,0,755,0,,1,4,2,,4,,2,0,,1,L7OK,301,0,0,3,1,0,0,0,0,,,,0,0,,,,,141,Moved Permanently,,0,1,8,8, application-backend,BACKEND,0,0,0,1,400,7,1292,19718,0,0,,0,0,0,0,UP,1,1,0,,0,755,0,,1,4,0,,7,,1,0,,1,,,,0,5,2,0,0,0,,,,,0,0,0,0,0,0,141,,,0,1,8,8,
For nginx:
#!/bin/bash set -e echo "info: Draining connections ..." nginx -s quit echo "info: You have 5 minutes to start and finish your upgrade." sleep 300 nginx -s start echo "info: nginx restarted!" exit 0
If you’re using a configuration management system, like puppet or Chef, you can remove the service from your load balancer pool. This works well in practice with only 2 or 3 servers, though draining is usually not considered.
Note that when using the popular “reverse HAProxy” setup with application servers running HAProxy on localhost, and HAProxy forwarding localhost requests to the real servers (like httpd), then you want to stop or block the httpd services on the real server end. Otherwise you would have to make changes on multiple application servers.
In a future post, I’ll discuss zero-downtime deploys.
Drain connections on restart of NGINX process? (with iptables)
Tomcat’s Graceful Shutdown with Daemons and Shutdown Hooks
Get haproxy stats/informations via socat
Go net/http: add built-in graceful shutdown support to Server #4674
haproxy.tech-notes.net: HAProxy Socket Commands