Alteryx automation Desktop (Scheduling)

Hey everyone,

I keep having the same error while scheduling a workflow (Remote Storage Error.)

This error only pops up when the workflow involves DBs (writing and/or reading) even though I configured the scheduler as this article showed https://knowledge.alteryx.com/index/s/article/Configure-Desktop-with-Scheduler-1583460917472 and added the run as a user.

The error exists whether I choose to run from disk or in Scheduler DB.

Could you please help me resolve this?

Thanks in advance

Alteryx Desktop automation error.png

Database Connection

Scheduler

Accepted answers

All comments

devinrahn

I am running into this problem as well. Very frustrating. These are workflows I was running on schedule for months and months without issue, and now they fail every time.

The workflows run perfectly fine outside of Scheduler. This is making Scheduler virtually useless.

apathetichell

turn off amp.

AT_AYX

Hello @devinrahn

I found out that there may be a problem with DCM (Data Connection Manager).

Please try to deselect the DCM and configure the connection with DSN/DSN-less because that is how it worked for me

AT_AYX

Hello @apathetichell ,

It didn't work for me.

Drvt6713

@AT_AYX

I found out from Support that the Scheduler is not compatible with any flows that use any of the new Python based tools such as connectors. Rendered the use of scheduler for most of my planned flows as incompatible. Not sure if this will help with your flow or not.

AT_AYX

@Drvt6713

Unfortunately, this is not my case; I only use input/output data tools to read/write to a DB.

But good to know, thank you !!

ShaneXue

Have you solved this issue? I encountered the same problem as well, whenever I use in-db tool, I get the same error as you. so sad!!

AT_AYX

Hey @ShaneXue,

Sorry for the late response, but did you try it without DCM?

BigBanimo

Hey!
I recently ran into this problem, and I disabled the DCM on my Input Data tools and used the standard connection string and it resolved the issue

EdP

Please see the following KB for this error:

https://knowledge.alteryx.com/index/s/article/Remote-storage-error

MartyScherr

So this article did not resolve my Alteryx Schedules throwing the Remote Storage Error

1. I am NOT using DCM. I'm using DSN connection strings. I'm also not using Python or other strange setup as suggested by some of the Knowledge base articles.

NOTE: This Schedule worked fine a few days ago so it can not be the workflow design that is the issue.

2. I'm using the Store on Disk Option in the schedule as I thought maybe it was an issue trying to store a large amount of data but that doesn't seem to be the case.

3. See attached screenshot for the the Output and Error Message I received. Please don't point me to an article. I'm looking for a practical fix to this issue.

apathetichell

This is timing out at 15 minutes. This makes me think some authorization/setting expires at 15 minutes. I don't know what's in your workflow - and I don't know any system settings you have - but it looks like something somewhere is set to timeout after 15 minutes. Perhaps this changed on your system a few days ago and something which used to be longer lived (like an authorization token) now expires in 15 minutes?

MartyScherr

So what is interesting is that the Workflow Schedule runs fine when I run it manually in Designer. It's only when I run it with WorkFlow Scheduler that it appears to timeout in 15 minutes. Still looking for a resolution on this. For Clarity: I'm using the Input Data and Output Data tools in this workflow. The Input Data is using a DSN connection string pointed at NetSuite Oracle ODBC. The Output Data tool is pointed at Azure SQL.

apathetichell

I'd ask the admins for those platforms what the timeout is for however you authenticate. If you previously could run this workflow on scheduler -> but now can't -you should ask if anything changed recently in terms of timeouts.

timeout may be totally different when you are running in desktop --> 1) are you monitoring the execution time, and is it under 15 minutes 2) are you (inadvertently) doing something which allows for a refresh. You're not using DCM - correct?

MartyScherr

I am not using DCM

I am not using the Amp Engine - which had apparently caused issues for some users running schedulers.

I don't think it's a Netsuite timeout issue because why would a manual run of the same workflow execute successfully using the same workflow?

I was able to get the workflow to execute successfully by using a Windows Scheduled Task using the Alteryx Command Line Utility. That's the option I am going to use for now.

Anyone else would may have experienced a similar experience with Scheduler timing out I would be interested in hearing about how you resolved it. Thank you.

apathetichell

@MartyScherr- @drewwoodall is correct that hardcoding credentials could effect the timeout. @drewwoodall -> ask what the timeout setting is on the AWS STS session.

Hypothesis:

IAM role is native to the system.

IAM role uses AWS STS to get short term credentials to access S3.

Credentials expire mid operation (say 15 minutes in)

Fixes:

1) extend STS credentials lifespan for this role.

2) redesign workflow to split data into 15 minute lots.

15 minutes of S3 upload seems like a decent chunk -> perhaps you can turn this into multiple async flows?

Note - I would use AWS CLI vs the Alteryx S3 tool/DCM because I know what it's doing and how to control it and I don't have to worry about code bloat.

drewwoodall

Hi thanks for the quick response, but these aren't temporary session based keys. These are the IAM Access Key and Secret key that have to be manually cycled by our admins to expire. So timeout really isn't a problem,

I've run download workflows taken 25+ hours without hitting a timeout. They only seem to work when we run workflows manually from designer to upload and download from our S3 buckets. Once we get scheduler involved we get hit with that remote storage error.

MartyScherr

So what I'm still trying to understand is why there is a timeout of 15 minutes when run the workflows from Scheduler via Desktop Automation and not when running from the command line utility?

drewwoodall

Talked to support and basically DCM and scheduler won't work together, if I want to do it with Alteryx I'm going to have to hard code the keys into each tool and between the raw and prepare steps I'm looking at needing about 300 of them. So that's not really an optimal solution.

Then I remembered what one of the AWS guys we've been talking to told me, whatever you can do in the console is basically an AWS CLI command. So it got me digging into the CP --recursive command and the Sync command. So since I've got a EC2 windows server w/ a network drive I'm going to be using to break up the workflows anyway, I'm looking into using task manager to just sync a formatted output straight up to the relevant S3 buckets & their folders. I wish Alteryx's output tool could create folders like the S3 upload tool can in buckets but apparently people have been asking for that since 2018 so looks like the mkdir will be the answer. Gotta have an individual folder for each unpartitioned avro file so that when it's crawled by glue it'll populate as a distinct table.

apathetichell

@drewwoodall-> returning to 'use the run command tool - and auth via aws cli - don't use keys.' There is no issue with using the run command tool on scheduler. and obviously there is more documentation and are more options to use the aws cli vs the alteryx connectors.

Quick Links

This months top contributors

atcodedog05 19458

Qiu 15865

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703