Download Tool - Error transferring data: Couldn't connect to server

[mpurdy]

I was wondering if anyone has experienced this error and if there were any common problems I could check, apart from the obvious one (that the server is actually unavailable)?

I am performing a POST with JSON to a REST API. The error occurs intermittently, occurs at different times during the day and sometimes in the middle of a batch (the first couple of records will process and then one will fail). So I am not sure if the server becomes unavailable due to processing the first couple of records or if the error would occur for something like a timeout?

Appreciate any ideas or feedback people could suggest.

[michael_treadwell]

It is tough to tell exactly why this would happen but my first guess would be this:

Most APIs have a call limit that only allows so many calls in so many number of seconds. If you exceed this, you will be blocked from making calls for a specified time frame.

It could be that. It could not. However, if it is truly a REST API, you should be able to look at the response headers that you get from the download tool. Instead of just looking at the [DownloadData] field from the download tool, try looking at the [DownloadHeaders] field for the HTTP reponse code and associated message. This may give you more insight to the problem.

[NeilR]

Check the API documentation for limitations. If for instance, you can only do 5 requests per minute, you could use the Throttle tool appropriately space the requests. If that doesn't work, you could try upping the Timout parameter in the Connection tab of the Download tool.

[mpurdy]

Thanks for the feedback and suggestions.

Unfortunately, because the connection to the server is failing, there isn't anything actually in the [DownloadData] regarding the failure. The connection has not even been made to POST the JSON to receive a response. Is there any kind of logging for the download tool which would give me detailed information on where the failure occurred?

There isn't really documentation for the API but you have reminded me that I was meaning to adjust the Maximum Connections to 1 and increase the timeout so as not to overload the API. The timeout says that it is in seconds, is that correct? It is set to 600 by default, that means that it is set to 10 minutes, but the job fails within a minute so I would assume that it is not timing out?

The other option that I would like to try would be to introduce a 500 millisecond delay between processing each record but I can't see an easy way to do this? Thanks again for all your help.

[NeilR]

Use the Throttle tool to introduce a delay between processing records.

[mpurdy]

Thanks for the suggestions. I don't think the throttle tool will help, unless I only process one or two records every minute, which is too slow. It would be good if the throttle tool could specify how many records to process per second (rather than minute) or per defined time interval.

[NeilR]

120 records per minute = 2 records per second

[NeilR · Accepted solution]

Perhaps the Throttle tool help file is a bit confusing. It does not send a batch of X records through each minute, but rather a single record per time increment. For example setting to 60 records per minute sends 1 record through per second. See screenshot below, where the Message tool after the Throttle tool prints out the datetime. @PaulaA

 

[mpurdy]

Thanks for clarifying. Yes I think the help documentation is a bit confusing as it does not specifically state that it will evenly space the record processing. It only states that it will process a certain number per minute with a minimum spacing of 1 millisecond, so I assumed that all processing could occur in the first 10 seconds and then it would wait 50 seconds before processing the next X records. If it could be fixed that would be good to avoid future confusion.

I will give the throttle tool a go and see if it helps. Thanks!