This site uses different types of cookies, including analytics and functional cookies (its own and from other sites). To change your cookie settings or find out more, click here. If you continue browsing our website, you accept these cookies.

Learn more
I AGREE
Reject

Early bird tickets for Inspire 2023 are now available! Discounted pricing closes on January 31st. Save your spot!

community Data Science Portal Alteryx.com
alteryx Community
Search
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Close
Free Trial

Alteryx Promote Knowledge Base

Definitive answers from Promote experts.

Supporting Promote Series Part 2 - Promote Services are Down

JohnPo
Alteryx Alumni (Retired)
Created

on ‎10-17-2018 08:49 AM - edited on ‎02-27-2020 01:29 PM by Alteryx Community Team

Welcome to part 2 of the Supporting Promote series. In this series, we will tackle some common issues and questions, and provide best practices for troubleshooting. In this article, we will be investigating one common "Promote Service Down" scenario - when the promote_logspout and promote_logstash services are down.  You can follow these steps if other services are down.

 

How can I check the statuses of the Promote services? 

 

To verify the status of the services, run the following command:

 

sudo promote.status

 

If a service returns a ratio in the Replicas field that does not resolve to 100% (e.g., “0/0”, “0/1”, “1/3”, “2/3”, etc.) this means that one or more of the Promote services are down.

 

promoteservicedown.png

 

How can I troubleshoot a downed service? 

 

This is an example of how to set up the command that will show you the logs for the service:

 

sudo docker service logs {Service Name}

 

Then, run the command for promote_logspout service.

 

sudo docker service logs promote_logspout

 

The results from the command above:

Spoiler
promote_logspout.0.xzsx3m6m6rpc@machine-03 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.wb94j4ls5bzs@machine-03 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.6p8zeyd4jhsf@machine-03 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.zq0nkf4yghma@machine-03 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.vfw9umulimf2@machine-03 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.ra7265z4hztw@machine-03 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.ra7265z4hztw@machine-03 | # adapters: tls udp logstash syslog raw tcp
promote_logspout.0.xzsx3m6m6rpc@machine-03 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.wb94j4ls5bzs@machine-03 | # adapters: raw tcp tls udp logstash syslog
promote_logspout.0.6p8zeyd4jhsf@machine-03 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.zq0nkf4yghma@machine-03 | # adapters: raw tcp tls udp logstash syslog
promote_logspout.0.vfw9umulimf2@machine-03 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.vfw9umulimf2@machine-03 | # options : persist:/mnt/routes
promote_logspout.0.ra7265z4hztw@machine-03 | # options : persist:/mnt/routes
promote_logspout.0.xzsx3m6m6rpc@machine-03 | # options : persist:/mnt/routes
promote_logspout.0.wb94j4ls5bzs@machine-03 | # options : persist:/mnt/routes
promote_logspout.0.6p8zeyd4jhsf@machine-03 | # options : persist:/mnt/routes
promote_logspout.0.zq0nkf4yghma@machine-03 | # options : persist:/mnt/routes
promote_logspout.0.zq0nkf4yghma@machine-03 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.vfw9umulimf2@machine-03 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.ra7265z4hztw@machine-03 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.xzsx3m6m6rpc@machine-03 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.wb94j4ls5bzs@machine-03 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.6p8zeyd4jhsf@machine-03 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.cmr173976iix@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.u33flr4wdvpn@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.s3jy3vn1rx01@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.wyfwo6urigsa@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.e8ge4dr5nvac@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.z6tfvk1d6bio@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.vesviqscr2jk@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.jm137d7ki41i@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.q9j1j36vnpas@machine-02 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.q9j1j36vnpas@machine-02 | # adapters: tls udp logstash syslog raw tcp
promote_logspout.0.e8ge4dr5nvac@machine-02 | # adapters: raw tcp tls udp logstash syslog
promote_logspout.0.z6tfvk1d6bio@machine-02 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.vesviqscr2jk@machine-02 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.jm137d7ki41i@machine-02 | # adapters: syslog raw tcp tls udp logstash
promote_logspout.0.wyfwo6urigsa@machine-02 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.u33flr4wdvpn@machine-02 | # adapters: syslog raw tcp tls udp logstash
promote_logspout.0.s3jy3vn1rx01@machine-02 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.vesviqscr2jk@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.q9j1j36vnpas@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.e8ge4dr5nvac@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.z6tfvk1d6bio@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.z6tfvk1d6bio@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.s3jy3vn1rx01@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.jm137d7ki41i@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.wyfwo6urigsa@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.u33flr4wdvpn@machine-02 | # options : persist:/mnt/routes
promote_logspout.0.q9j1j36vnpas@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.vesviqscr2jk@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.e8ge4dr5nvac@machine-02 | # jobs : routes http[]:80 pump
promote_logspout.0.jm137d7ki41i@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.s3jy3vn1rx01@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.wyfwo6urigsa@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.u33flr4wdvpn@machine-02 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.e8ge4dr5nvac@machine-02 | # routes :
promote_logspout.0.e8ge4dr5nvac@machine-02 | # ADAPTER ADDRESS CONTAINERS SOURCES OPTIONS
promote_logspout.0.e8ge4dr5nvac@machine-02 | # logstash logstash:5000 map[]
promote_logspout.0.e8ge4dr5nvac@machine-02 | 2018/07/09 13:03:20 logstash: could not write:write udp 10.0.0.13:41704->10.0.0.8:5000: write: operation not permitted
promote_logspout.0.sip6bqk7hbm1@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.y4jukhhr7s3j@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.q1idkpi3eu5r@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.n5w3bkuzx73j@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.k9o89k7tyu17@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.kxyq68t7et46@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.juhql4zlg7m8@machine-01 | # logspout v3.2.4-custom by gliderlabs
promote_logspout.0.juhql4zlg7m8@machine-01 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.q1idkpi3eu5r@machine-01 | # adapters: tls udp logstash syslog raw tcp
promote_logspout.0.n5w3bkuzx73j@machine-01 | # adapters: raw tcp tls udp logstash syslog
promote_logspout.0.cmr173976iix@machine-01 | # adapters: logstash syslog raw tcp tls udp
promote_logspout.0.sip6bqk7hbm1@machine-01 | # adapters: udp logstash syslog raw tcp tls
promote_logspout.0.y4jukhhr7s3j@machine-01 | # adapters: logstash syslog raw tcp tls udp
promote_logspout.0.k9o89k7tyu17@machine-01 | # adapters: syslog raw tcp tls udp logstash
promote_logspout.0.kxyq68t7et46@machine-01 | # adapters: raw tcp tls udp logstash syslog
promote_logspout.0.kxyq68t7et46@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.sip6bqk7hbm1@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.cmr173976iix@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.y4jukhhr7s3j@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.juhql4zlg7m8@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.q1idkpi3eu5r@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.n5w3bkuzx73j@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.k9o89k7tyu17@machine-01 | # options : persist:/mnt/routes
promote_logspout.0.y4jukhhr7s3j@machine-01 | # jobs : http[]:80 pump routes
promote_logspout.0.kxyq68t7et46@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.sip6bqk7hbm1@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.cmr173976iix@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.k9o89k7tyu17@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.y4jukhhr7s3j@machine-01 | # routes :
promote_logspout.0.juhql4zlg7m8@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.q1idkpi3eu5r@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.n5w3bkuzx73j@machine-01 | !! lookup logstash on 127.0.0.11:53: server misbehaving
promote_logspout.0.y4jukhhr7s3j@machine-01 | # ADAPTER ADDRESS CONTAINERS SOURCES OPTIONS
promote_logspout.0.y4jukhhr7s3j@machine-01 | # logstash logstash:5000 map[]
promote_logspout.0.y4jukhhr7s3j@machine-01 | 2018/07/09 13:03:20 logstash: could not write:write udp 10.0.0.20:51934->10.0.0.8:5000: write: operation not permitted

In this example, it seems that the promote_logspout service is misbehaving because promote_logstash service is also misbehaving. To fix this problem, restart both services, starting with promote_logstash, following with promote_logspout.

 

If you're going to update a service, then it's recommended to use the --detach  

 

sudo docker service update promote_logstash --detach=false --force

 

The results from the command above:

Spoiler
promote_logstash
overall progress: 1 out of 1 tasks
1/1: running [==================================================>]
verify: Service converged
 
Next, update the promote_logpsout service:
sudo docker service update promote_logspout --detach=false --force
 
The results from the command above:
Spoiler
promote_logspout
overall progress: 3 out of 3 tasks
dzys60hss1jg: running [==================================================>]
6iscs5yrjgo1: running [==================================================>]
qx4cu0k4hnfj: running [==================================================>]
verify: Service converged
 
Next, let's verify that all the services are up and running successfully.
promote.status
 
The results from the command above:
Spoiler
ID NAME MODE REPLICAS IMAGE PORTS
5zv8fa8p98zc promote_portainer replicated 1/1 portainer/portainer:latest *:9000->9000/tcp
7xx1lknbtbm8 promote_consul replicated 1/1 quay.io/yhat/consul:latest *:8500->8500/tcp
c119upi2o4a9 promote_registry replicated 1/1 registry:2 *:5000->5000/tcp
d2ou18daqy6e promote_logspout global 3/3 quay.io/yhat/logspout-logstash:latest
dpa1hvfvd0tl promote_elasticsearch replicated 1/1 quay.io/yhat/elasticsearch:latest *:9200->9200/tcp,*:9300->9300/tcp
i6lnee4nzkwu promote_registrator global 3/3 quay.io/yhat/registrator:latest
nmv4hc5w79t0 promote_app replicated 1/1 quay.io/yhat/promote-app:latest *:3001->3001/tcp
nwr1u85cj2i8 promote_nginx global 3/3 quay.io/yhat/nginx-proxy:latest
sipqdlsohckj promote_db replicated 1/1 quay.io/yhat/promote-db:latest *:5432->5432/tcp
uvxilx9tq45f promote_telegraf global 3/3 quay.io/yhat/telegraf:latest
whetelsj8sm9 promote_kibana replicated 1/1 quay.io/yhat/kibana:latest *:85->5601/tcp
woenbqwtfxzo promote_influxdb replicated 1/1 quay.io/yhat/influxdb:latest *:8086->8086/tcp
xvbvovhea02j promote_logstash replicated 1/1 quay.io/yhat/logstash:latest *:5001->5000/udp,*:9600->9600/tcp
zrblzuh9da1b promote_docker-events-mon global 3/3 quay.io/yhat/docker-events-mon:latest

Everything looks great!  Keep calm and Promote on!