Problem with Alteryx and FRED API

[tuomassalminen]

Hi,

I have tried to download large amount of macro economic and industrial indicators from FRED (https://fred.stlouisfed.org/) Economic Research of St. Louis FED. My workflow is basic, just creating bunch of URL:s in the right format for the API, feeding those to the download tool and lastly parsing the XML data to a more usable format.

Everything works well with 50-100ish data series but for larger amounts (200-1000 series) the workflow returns null values. Has someone used this particular API and knows if there is some sort of hard cap for amount of requests per workflow? Has anyone noticed similar issues when dealing with other (REST) APIs?

[grossal]

 

Hi @tuomassalminen,

I don't know FRED, but sometimes it helps if you add a user-agent to the download tool. 

 (You can find out the user agent of your browser by googling it or just copy mine)

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36

The API documentation just mentions "1000 records per request" as a cap, but nothing else.

Let me know if the User-Agent helped.

Best

Alex

[danilang]

Hi  @tuomassalminen 

From looking at the FRED api documentation for downloading observations, it looks like there's a limit of 100000 per request. 

To request the next group use the offset=100001 parameter

Dan

[tuomassalminen]

Hi,

Thank you both for the replies. I will try adding the user-agent just in case. It seems like there's clear limitations on the size of the requests (how many observations) and frequency of requests. I will try again by tweaking the parameters and will follow-up.

I also received an answer from FRED. They responded by saying that the API has a rate limit of 120 request per minute.

[bmack]

@danilang

On this subject, I'm having trouble with the URL setup and where to input my API

 You can see that no data is downloaded.

THanks

[apathetichell]

@bmack- attach a text to columns tool. split that field to rows. use /n as your delimiter.

[matthew_foster]

I encountered the same issue.  I placed a Throttle tool just before the download tool and throttled to 100 records per minute. Each record in my case was 1 FRED Series, so this kept me under the 120 series per minute limit that FRED allows.