This site uses different types of cookies, including analytics and functional cookies (its own and from other sites). To change your cookie settings or find out more, click here. If you continue browsing our website, you accept these cookies.
Early bird tickets for Inspire 2023 are now available! Discounted pricing closes on January 31st. Save your spot!
on
10-17-2018
08:49 AM
- edited on
02-27-2020
01:29 PM
by
KylieF
Welcome to part 2 of the Supporting Promote series. In this series, we will tackle some common issues and questions, and provide best practices for troubleshooting. In this article, we will be investigating one common "Promote Service Down" scenario - when the promote_logspout and promote_logstash services are down. You can follow these steps if other services are down.
How can I check the statuses of the Promote services?
To verify the status of the services, run the following command:
sudo promote.status
If a service returns a ratio in the Replicas field that does not resolve to 100% (e.g., “0/0”, “0/1”, “1/3”, “2/3”, etc.) this means that one or more of the Promote services are down.
How can I troubleshoot a downed service?
This is an example of how to set up the command that will show you the logs for the service:
sudo docker service logs {Service Name}
Then, run the command for promote_logspout service.
sudo docker service logs promote_logspout
The results from the command above:
promote_logspout.0.xzsx3m6m6rpc@machine-03 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.wb94j4ls5bzs@machine-03 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.6p8zeyd4jhsf@machine-03 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.zq0nkf4yghma@machine-03 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.vfw9umulimf2@machine-03 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.ra7265z4hztw@machine-03 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.ra7265z4hztw@machine-03 | # adapters: tls udp logstash syslog raw tcp
promote_logspout.0.xzsx3m6m6rpc@machine-03 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.wb94j4ls5bzs@machine-03 | # adapters: raw tcp tls udp logstash syslog
promote_logspout.0.6p8zeyd4jhsf@machine-03 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.zq0nkf4yghma@machine-03 | # adapters: raw tcp tls udp logstash syslog
promote_logspout.0.vfw9umulimf2@machine-03 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.vfw9umulimf2@machine-03 | # options : persist:/mnt/routes
promote_logspout.0.ra7265z4hztw@machine-03 | # options : persist:/mnt/routes
promote_logspout.0.xzsx3m6m6rpc@machine-03 | # options : persist:/mnt/routes
promote_logspout.0.wb94j4ls5bzs@machine-03 | # options : persist:/mnt/routes
promote_logspout.0.6p8zeyd4jhsf@machine-03 | # options : persist:/mnt/routes
promote_logspout.0.zq0nkf4yghma@machine-03 | # options : persist:/mnt/routes
promote_logspout.0.zq0nkf4yghma@machine-03 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.vfw9umulimf2@machine-03 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.ra7265z4hztw@machine-03 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.xzsx3m6m6rpc@machine-03 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.wb94j4ls5bzs@machine-03 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.6p8zeyd4jhsf@machine-03 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.cmr173976iix@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.u33flr4wdvpn@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.s3jy3vn1rx01@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.wyfwo6urigsa@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.e8ge4dr5nvac@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.z6tfvk1d6bio@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.vesviqscr2jk@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.jm137d7ki41i@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.q9j1j36vnpas@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.q9j1j36vnpas@machine-02 | # adapters: tls udp logstash syslog raw tcp
promote_logspout.0.e8ge4dr5nvac@machine-02 | # adapters: raw tcp tls udp logstash syslog
promote_logspout.0.z6tfvk1d6bio@machine-02 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.vesviqscr2jk@machine-02 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.jm137d7ki41i@machine-02 | # adapters: syslog raw tcp tls udp logstash
promote_logspout.0.wyfwo6urigsa@machine-02 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.u33flr4wdvpn@machine-02 | # adapters: syslog raw tcp tls udp logstash
promote_logspout.0.s3jy3vn1rx01@machine-02 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.vesviqscr2jk@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.q9j1j36vnpas@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.e8ge4dr5nvac@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.z6tfvk1d6bio@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.z6tfvk1d6bio@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.s3jy3vn1rx01@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.jm137d7ki41i@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.wyfwo6urigsa@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.u33flr4wdvpn@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.q9j1j36vnpas@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.vesviqscr2jk@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.e8ge4dr5nvac@machine-02 | # jobs : routes http[]:80 pump
promote_logspout.0.jm137d7ki41i@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.s3jy3vn1rx01@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.wyfwo6urigsa@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.u33flr4wdvpn@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.e8ge4dr5nvac@machine-02 | # routes :
promote_logspout.0.e8ge4dr5nvac@machine-02 | # ADAPTER ADDRESS CONTAINERS SOURCES OPTIONS
promote_logspout.0.e8ge4dr5nvac@machine-02 | # logstash logstash:5000 map[]
promote_logspout.0.e8ge4dr5nvac@machine-02 | 2018/07/09 13:03:20 logstash: could not write:write udp 10.0.0.13:41704->10.0.0.8:5000: write: operation not permitted
promote_logspout.0.sip6bqk7hbm1@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.y4jukhhr7s3j@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.q1idkpi3eu5r@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.n5w3bkuzx73j@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.k9o89k7tyu17@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.kxyq68t7et46@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.juhql4zlg7m8@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.juhql4zlg7m8@machine-01 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.q1idkpi3eu5r@machine-01 | # adapters: tls udp logstash syslog raw tcp
promote_logspout.0.n5w3bkuzx73j@machine-01 | # adapters: raw tcp tls udp logstash syslog
promote_logspout.0.cmr173976iix@machine-01 | # adapters: logstash syslog raw tcp tls udp
promote_logspout.0.sip6bqk7hbm1@machine-01 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.y4jukhhr7s3j@machine-01 | # adapters: logstash syslog raw tcp tls udp
promote_logspout.0.k9o89k7tyu17@machine-01 | # adapters: syslog raw tcp tls udp logstash
promote_logspout.0.kxyq68t7et46@machine-01 | # adapters: raw tcp tls udp logstash syslog
promote_logspout.0.kxyq68t7et46@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.sip6bqk7hbm1@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.cmr173976iix@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.y4jukhhr7s3j@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.juhql4zlg7m8@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.q1idkpi3eu5r@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.n5w3bkuzx73j@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.k9o89k7tyu17@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.y4jukhhr7s3j@machine-01 | # jobs : http[]:80 pump routes
promote_logspout.0.kxyq68t7et46@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.sip6bqk7hbm1@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.cmr173976iix@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.k9o89k7tyu17@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.y4jukhhr7s3j@machine-01 | # routes :
promote_logspout.0.juhql4zlg7m8@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.q1idkpi3eu5r@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.n5w3bkuzx73j@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.y4jukhhr7s3j@machine-01 | # ADAPTER ADDRESS CONTAINERS SOURCES OPTIONS
promote_logspout.0.y4jukhhr7s3j@machine-01 | # logstash logstash:5000 map[]
promote_logspout.0.y4jukhhr7s3j@machine-01 | 2018/07/09 13:03:20 logstash: could not write:write udp 10.0.0.20:51934->10.0.0.8:5000: write: operation not permitted
In this example, it seems that the promote_logspout service is misbehaving because promote_logstash service is also misbehaving. To fix this problem, restart both services, starting with promote_logstash, following with promote_logspout.
If you're going to update a service, then it's recommended to use the --detach
sudo docker service update promote_logstash --detach=false --force
The results from the command above:
promote_logstash overall progress: 1 out of 1 tasks 1/1: running [==================================================>] verify: Service converged
sudo docker service update promote_logspout --detach=false --force
promote_logspout overall progress: 3 out of 3 tasks dzys60hss1jg: running [==================================================>] 6iscs5yrjgo1: running [==================================================>] qx4cu0k4hnfj: running [==================================================>] verify: Service converged
promote.status
ID NAME MODE REPLICAS IMAGE PORTS 5zv8fa8p98zc promote_portainer replicated 1/1 portainer/portainer:latest *:9000->9000/tcp 7xx1lknbtbm8 promote_consul replicated 1/1 quay.io/yhat/consul:latest *:8500->8500/tcp c119upi2o4a9 promote_registry replicated 1/1 registry:2 *:5000->5000/tcp d2ou18daqy6e promote_logspout global 3/3 quay.io/yhat/logspout-logstash:latest dpa1hvfvd0tl promote_elasticsearch replicated 1/1 quay.io/yhat/elasticsearch:latest *:9200->9200/tcp,*:9300->9300/tcp i6lnee4nzkwu promote_registrator global 3/3 quay.io/yhat/registrator:latest nmv4hc5w79t0 promote_app replicated 1/1 quay.io/yhat/promote-app:latest *:3001->3001/tcp nwr1u85cj2i8 promote_nginx global 3/3 quay.io/yhat/nginx-proxy:latest sipqdlsohckj promote_db replicated 1/1 quay.io/yhat/promote-db:latest *:5432->5432/tcp uvxilx9tq45f promote_telegraf global 3/3 quay.io/yhat/telegraf:latest whetelsj8sm9 promote_kibana replicated 1/1 quay.io/yhat/kibana:latest *:85->5601/tcp woenbqwtfxzo promote_influxdb replicated 1/1 quay.io/yhat/influxdb:latest *:8086->8086/tcp xvbvovhea02j promote_logstash replicated 1/1 quay.io/yhat/logstash:latest *:5001->5000/udp,*:9600->9600/tcp zrblzuh9da1b promote_docker-events-mon global 3/3 quay.io/yhat/docker-events-mon:latest
Everything looks great! Keep calm and Promote on!