Supporting Promote Series Part 2 - Promote Services are Down

[JohnPo]

Welcome to part 2 of the Supporting Promote series. In this series, we will tackle some common issues and questions, and provide best practices for troubleshooting. In this article, we will be investigating one common "Promote Service Down" scenario - when the promote_logspout and promote_logstash services are down.  You can follow these steps if other services are down.

How can I check the statuses of the Promote services? 

To verify the status of the services, run the following command:

sudo promote.status

If a service returns a ratio in the Replicas field that does not resolve to 100% (e.g., “0/0”, “0/1”, “1/3”, “2/3”, etc.) this means that one or more of the Promote services are down.

How can I troubleshoot a downed service? 

This is an example of how to set up the command that will show you the logs for the service:

sudo docker service logs {Service Name}

Then, run the command for promote_logspout service.

sudo docker service logs promote_logspout

The results from the command above:

Spoiler

promote_logspout.0.xzsx3m6m6rpc@machine-03 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.wb94j4ls5bzs@machine-03 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.6p8zeyd4jhsf@machine-03 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.zq0nkf4yghma@machine-03 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.vfw9umulimf2@machine-03 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.ra7265z4hztw@machine-03 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.ra7265z4hztw@machine-03 | # adapters: tls udp logstash syslog raw tcp
promote_logspout.0.xzsx3m6m6rpc@machine-03 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.wb94j4ls5bzs@machine-03 | # adapters: raw tcp tls udp logstash syslog
promote_logspout.0.6p8zeyd4jhsf@machine-03 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.zq0nkf4yghma@machine-03 | # adapters: raw tcp tls udp logstash syslog
promote_logspout.0.vfw9umulimf2@machine-03 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.vfw9umulimf2@machine-03 | # options : persist:/mnt/routes
promote_logspout.0.ra7265z4hztw@machine-03 | # options : persist:/mnt/routes
promote_logspout.0.xzsx3m6m6rpc@machine-03 | # options : persist:/mnt/routes
promote_logspout.0.wb94j4ls5bzs@machine-03 | # options : persist:/mnt/routes
promote_logspout.0.6p8zeyd4jhsf@machine-03 | # options : persist:/mnt/routes
promote_logspout.0.zq0nkf4yghma@machine-03 | # options : persist:/mnt/routes
promote_logspout.0.zq0nkf4yghma@machine-03 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.vfw9umulimf2@machine-03 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.ra7265z4hztw@machine-03 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.xzsx3m6m6rpc@machine-03 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.wb94j4ls5bzs@machine-03 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.6p8zeyd4jhsf@machine-03 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.cmr173976iix@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.u33flr4wdvpn@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.s3jy3vn1rx01@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.wyfwo6urigsa@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.e8ge4dr5nvac@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.z6tfvk1d6bio@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.vesviqscr2jk@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.jm137d7ki41i@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.q9j1j36vnpas@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.q9j1j36vnpas@machine-02 | # adapters: tls udp logstash syslog raw tcp
promote_logspout.0.e8ge4dr5nvac@machine-02 | # adapters: raw tcp tls udp logstash syslog
promote_logspout.0.z6tfvk1d6bio@machine-02 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.vesviqscr2jk@machine-02 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.jm137d7ki41i@machine-02 | # adapters: syslog raw tcp tls udp logstash
promote_logspout.0.wyfwo6urigsa@machine-02 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.u33flr4wdvpn@machine-02 | # adapters: syslog raw tcp tls udp logstash
promote_logspout.0.s3jy3vn1rx01@machine-02 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.vesviqscr2jk@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.q9j1j36vnpas@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.e8ge4dr5nvac@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.z6tfvk1d6bio@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.z6tfvk1d6bio@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.s3jy3vn1rx01@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.jm137d7ki41i@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.wyfwo6urigsa@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.u33flr4wdvpn@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.q9j1j36vnpas@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.vesviqscr2jk@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.e8ge4dr5nvac@machine-02 | # jobs : routes http[]:80 pump
promote_logspout.0.jm137d7ki41i@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.s3jy3vn1rx01@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.wyfwo6urigsa@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.u33flr4wdvpn@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.e8ge4dr5nvac@machine-02 | # routes :
promote_logspout.0.e8ge4dr5nvac@machine-02 | # ADAPTER ADDRESS CONTAINERS SOURCES OPTIONS
promote_logspout.0.e8ge4dr5nvac@machine-02 | # logstash logstash:5000 map[]
promote_logspout.0.e8ge4dr5nvac@machine-02 | 2018/07/09 13:03:20 logstash: could not write:write udp 10.0.0.13:41704->10.0.0.8:5000: write: operation not permitted
promote_logspout.0.sip6bqk7hbm1@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.y4jukhhr7s3j@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.q1idkpi3eu5r@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.n5w3bkuzx73j@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.k9o89k7tyu17@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.kxyq68t7et46@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.juhql4zlg7m8@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.juhql4zlg7m8@machine-01 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.q1idkpi3eu5r@machine-01 | # adapters: tls udp logstash syslog raw tcp
promote_logspout.0.n5w3bkuzx73j@machine-01 | # adapters: raw tcp tls udp logstash syslog
promote_logspout.0.cmr173976iix@machine-01 | # adapters: logstash syslog raw tcp tls udp
promote_logspout.0.sip6bqk7hbm1@machine-01 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.y4jukhhr7s3j@machine-01 | # adapters: logstash syslog raw tcp tls udp
promote_logspout.0.k9o89k7tyu17@machine-01 | # adapters: syslog raw tcp tls udp logstash
promote_logspout.0.kxyq68t7et46@machine-01 | # adapters: raw tcp tls udp logstash syslog
promote_logspout.0.kxyq68t7et46@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.sip6bqk7hbm1@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.cmr173976iix@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.y4jukhhr7s3j@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.juhql4zlg7m8@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.q1idkpi3eu5r@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.n5w3bkuzx73j@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.k9o89k7tyu17@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.y4jukhhr7s3j@machine-01 | # jobs : http[]:80 pump routes
promote_logspout.0.kxyq68t7et46@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.sip6bqk7hbm1@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.cmr173976iix@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.k9o89k7tyu17@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.y4jukhhr7s3j@machine-01 | # routes :
promote_logspout.0.juhql4zlg7m8@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.q1idkpi3eu5r@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.n5w3bkuzx73j@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.y4jukhhr7s3j@machine-01 | # ADAPTER ADDRESS CONTAINERS SOURCES OPTIONS
promote_logspout.0.y4jukhhr7s3j@machine-01 | # logstash logstash:5000 map[]
promote_logspout.0.y4jukhhr7s3j@machine-01 | 2018/07/09 13:03:20 logstash: could not write:write udp 10.0.0.20:51934->10.0.0.8:5000: write: operation not permitted

In this example, it seems that the promote_logspout service is misbehaving because promote_logstash service is also misbehaving. To fix this problem, restart both services, starting with promote_logstash, following with promote_logspout.

If you're going to update a service, then it's recommended to use the --detach  

sudo docker service update promote_logstash --detach=false --force

The results from the command above:

Spoiler

promote_logstash
overall progress: 1 out of 1 tasks
1/1: running [==================================================>]
verify: Service converged

Next, update the promote_logpsout service:

sudo docker service update promote_logspout --detach=false --force

The results from the command above:

Spoiler

promote_logspout
overall progress: 3 out of 3 tasks
dzys60hss1jg: running [==================================================>]
6iscs5yrjgo1: running [==================================================>]
qx4cu0k4hnfj: running [==================================================>]
verify: Service converged

Next, let's verify that all the services are up and running successfully.

promote.status

The results from the command above:

Spoiler

ID NAME MODE REPLICAS IMAGE PORTS
5zv8fa8p98zc promote_portainer replicated 1/1 portainer/portainer:latest *:9000->9000/tcp
7xx1lknbtbm8 promote_consul replicated 1/1 quay.io/yhat/consul:latest *:8500->8500/tcp
c119upi2o4a9 promote_registry replicated 1/1 registry:2 *:5000->5000/tcp
d2ou18daqy6e promote_logspout global 3/3 quay.io/yhat/logspout-logstash:latest
dpa1hvfvd0tl promote_elasticsearch replicated 1/1 quay.io/yhat/elasticsearch:latest *:9200->9200/tcp,*:9300->9300/tcp
i6lnee4nzkwu promote_registrator global 3/3 quay.io/yhat/registrator:latest
nmv4hc5w79t0 promote_app replicated 1/1 quay.io/yhat/promote-app:latest *:3001->3001/tcp
nwr1u85cj2i8 promote_nginx global 3/3 quay.io/yhat/nginx-proxy:latest
sipqdlsohckj promote_db replicated 1/1 quay.io/yhat/promote-db:latest *:5432->5432/tcp
uvxilx9tq45f promote_telegraf global 3/3 quay.io/yhat/telegraf:latest
whetelsj8sm9 promote_kibana replicated 1/1 quay.io/yhat/kibana:latest *:85->5601/tcp
woenbqwtfxzo promote_influxdb replicated 1/1 quay.io/yhat/influxdb:latest *:8086->8086/tcp
xvbvovhea02j promote_logstash replicated 1/1 quay.io/yhat/logstash:latest *:5001->5000/udp,*:9600->9600/tcp
zrblzuh9da1b promote_docker-events-mon global 3/3 quay.io/yhat/docker-events-mon:latest

Everything looks great!  Keep calm and Promote on!