Method for paging REST API results

[GreggD]

I have over 30,000 records I need to retrieve via my Rest API call, however my REST API limits the number of records we can return to 1,000 records. SO I need create a paging solution. In a regular procedural language I would use control flow logic like a WHILE loop to control pulling 900 records at a time, until I reach the last record and then stop. Is there a way to accomplish the same thing in Alteryx? Thanks in advance!

[DiganP]

@GreggD This article posted earlier talks about this, please take a look! 

https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Iterative-API-call-with-pagination-amp...

You would have to use the iteration macro. You have to use the Interface Designer to set the loop. Part of the look will be the results and the 2nd part would be to create a string of the next url. Attached is a sample workflow that I like to show to clients around this use case.

[GreggD]

Thanks for the reply. I am using version 2018.4 of Alteryx and am not able to load your workflow since it is a newer version. I also could not find the Iteration Macro in Alteryx Designer or in the Gallery. Can you point me to where I can find it? Thanks.

[cadyb]

Hi Greg, you can change the version of the workflows by right clicking the yxmd files (after unpackaging the yxzp) and editing with a text editor like Notepad or Notepad++. Then change the yxmdVer on the second line to 2018.4 and save the file. You will need to turn the process into either an Iterative Macro or a Batch Macro to download the data from each of the pages. Check out the link to the article that Digan included in his previous reply!

Cady

[GreggD]

Sorry Cady, I am really a newbie with Alteryx. The "Web Scraiping with Pagination (1).yxzp workflow does not appear to be a traditional zip file, so I am not sure how to unpack it like you said to do. Is there a special tool to unpack a .yxzp file? My windows zip application does not recognize it.

[cadyb]

No problem! The yxzp is an Alteryx packaged workflow similar to a zip. Save the yxzp somewhere in your documents so that it's not in your downloads folder. When you open it, it should open in Alteryx and ask you a few questions - you can just hit yes, Import, yes, etc. Then you'll probably see the message that it was created in a newer version Alteryx. That's okay. Go back to the folder where you saved the yxzp and you will see a new folder in there that contains the workflows and contents. There you can change the workflow version of the .yxmd and you will need to open the macros folder and update the macro Web Scraping.yxmc version as well. 

Then you can open the yxmd and run the example. Since it does include the Macro - you can open it to look at how it's made by right clicking on the Web Scraping macro in the workflow and choosing Open Macro.

I just updated the versions to 2018.4 for you in this attachment so you should be able to open it now.

[GreggD]

Thanks Cady for your help. I was able successfully load the workflow after following your instructions. Digan, When I loaded the workflow, it said that I was missing macros. Once loaded, the workflow only had a Text Input tool and a ScreenScraper macro with no connecting lines. Was there supposed to be another macro? You had mentioned that I should use the Iteration Macro, however I have not been able to find that macro anywhere including in the Alteryx Gallery. So I am still stuck. Any guidance, such as where I can find the Iteration Macro, would be appreciated. Thanks, Gregg

[DiganP]

@GreggD Attached is the macro with 2018.4 version changed! Let me know if this works!

[GreggD]

Thanks Digan. That worked better and loaded fine. This will give me something to study. My cooworked just found an article that actually explains what the Iteration tool is.... Apparently it is not a regular tool that I have on my tool bar, but something else entirely. So this explains why I could not find it anywhere. Here is the link to the article. So now I have some studying to do. https://community.alteryx.com/t5/Engine-Works-Blog/Hello-Iterative-Macro-My-Old-Friend/ba-p/420308 I will report back to you what I learn. Thanks, Gregg

[GreggD]

Ok, after reviewing the article I tried following it, however I am struggling with understanding how things need to be setup and configured. Although the article is a great start. It assumes some level of Alteryx knowledge that I don't have. I followed the part about changing the configuration of the workflow tab to select Macro: and then change the selection to Iterative Macro. However, how it explained to get to the Interface Designer did not work for me. Nothing was showing. I had to hunt it down and finally found it under the View Menu Option at the top of AlteryxDesigner. Then when I got it open and click on the Iteration Input field, it was a pulldown with nothing in it. Is it safe for me to assume that Interface Designer is looking for something specific to populate the Iteration Input and Iteration Output fields? I have more questions, however I think it would be better to take this a step at a time. I have attached two screen shots...at least I tried attaching....hoped it worked. One is a screenshot of my Interface Designer configuration issue with the empty pulldowns and then a screenshot of my workflow.