Download Tool - API Question (POST Action)

[DanielG]

Hi,

I have been using Alteryx for a while, but am new to the world of APIs and using the Download tool to interact with an API.  

I have datasets of about 2000-3000 records that I need to "send up" but I have set it up to run each row individually rather than create 30 payloads of 100 records per.  

With that I am getting inconsistent runs where the majority of time it handles all the rows, other times it only returns some of them.  No real errors in the Alteryx process that I can see.  It appears to solely be that when the API returns less records it throws "401 unauthorized on those rows".  That doesnt make sense because all of the other "OK" records use the same key and authorization.  And sometimes the runs are completely success and those same rows come bake "OK".

Does anyone have any thoughts?  Or can someone just confirm if I should/shouldnt be pushing each row individually?  

Within the basic tab of the download tool, my text string "URL" field is picked for the URL field.  And the checkbox for Encode URL text IS checked.

In headers, I load the two headers (as constants) that show in the samples from the URL syntax within the API documentation.  No boxes are checked below.

In the payload I select the "Take Query String/Body from Field" which is a field that I took sample of the payload from URL documentation and I replace the placeholders with my pertinent data fields.

I did not alter anything in the Connection tab, so it shows the name/pw as empty and the last 2 boxes show max connections = 2 and timeout = 600

I apologize for my lack of API knowledge, hopefully I explained this well enough to be understood.  If not, please let me know what I am missing.

[DanielG]

Thanks.  CASS Tool is great.  Used it for years.  Unfortunately it is no longer an option for me.  That is why I am working on this API.

I have a inquiry at the API's customer support if they can figure out if it is something wrong on their end.  Every from the Alteryx side seems clean, so I wonder if I am just hitting too much and too quickly.

I have this as a batch macro to pass in small groupings of about 20 records per payload, but I still inconsistently get the 401 error. 

I will post as I learn more, as I still dont know where the error actually lives and whether it is something wrong with my workflow versus an issue with the API itself.

Thanks for all the information on here.  It has been extremely helpful.

[BrandonB]

Of course! Happy to help with any other followups. Maybe they can check the logs on their side to see if anything isn't coming through properly.

One other tool that might be useful for you is the Throttle tool: https://help.alteryx.com/current/designer/throttle-tool

"Use Throttle to slow the speed of downstream processing by limiting the number of records that are passed through it. This is useful for specifying the number of requests sent per minute when there are limits to how many records can be processed.

Use the Throttle tool when sending or retrieving data from the web where requests have a rate limit. The Throttle tool will slow down the runtime of the workflow, instituting a wait for a number of records to pass before processing resumes."

If they are blocking your attempts at calling the API due to the frequency of the API calls, you might need to use the throttle tool before the download so that each call is spaced out enough so that it doesn't hit their request limits. My only other thought is that maybe one or more of your team members could also be calling the API using that account which could hit their limits. The 401 unauthorized message is puzzling though... normally that would be credential related so it is very odd that it would work sometimes but not others. 

[DanielG]

@BrandonB 

Happy New Year.  Hope all is well with you.

I was wondering if you could provide some additional information on this one from last month.  I still get the intermittent errors on this, so I took your advice and contacted their Customer Support.  While watching the logs as I was running the workflow, they are seeing successful receipt of my payload and successful transmission of the output back to me even though the workflow on my computer is not giving results and showing a 401 error. 

Do you have any idea why/how that could happen?

Could it be something as simple as a network "hiccup" causing a brief disconnect?  Or something else like that?

Also when I look in the download tool output on this error the error code in the DownloadHeaders field is the 401 Unauthorized, but when you look in the DownloadData field there is some HTML and it has Title: Access Denied, which is apparently a 403 error, not a 401 error (According to the customer support I talked to).

Anything additional you provide on this topic would be great.

Thanks
Dan

[BrandonB]

Happy New Year! The biggest difference between 401 and 403 is that 401 means the authorization was refused for the credentials provided while a 403 means that the server understood the request but refused to fulfill it. This means that the client is known by the server but it isn't sending the data so it sounds like the API is briefly telling you that your credentials are trying to access restricted content. This sounds like an issue with their service and/or a network connection to it: 

https://smartystreets.com/docs/cloud/requirements#retry

It would appear that they ask their users to build in a "retry" mechanism. I guess one approach would be to turn your call into an iterative macro and repeat the call if the status contains 403. That way if it ever hit that status the call would "loop" again and see if it could re-execute with a successful status the next time. 

Here is a training video on iterative macros if you haven't built one before: https://community.alteryx.com/t5/Videos/Build-Your-First-Iterative-Macro/td-p/55565

This may be your best bet if that is the case, although I'm still not sure why you would be returning a 403 if you have access to the information. 

[DanielG]

Thanks @BrandonB.  I will give it a shot.  I have experience with macros, but most have been of the batch variety.

I will update this post with the outcome soon.