Iterative macro for API pagination retrieval

[MarkEugene9000]

Good morning.

I am trying to pull down data via API and found that the API returns data for about 100 records at a time. There are about 600+ in my dummy data source. The bottom of the API results provide a 'next page' URL which I am trying to re-process through the workflow via an interative macro.

After trying to replicate some examples seen in this community I have been able to isolate the 'next page' URL at the end of the initial results, however, I am not quite sure how to get it to get sent back to to the beginning of an iterative macro. The initial data input is the initial API URL, token and user ID. I believe I need to just replace the URL with the 'next page' URL each time.

I attached my workflow and macro, and an image of the macro itself is shown below to try to illustrate what I'm doing.

This is my first macro attempt so forgive me if I'm missing something fundamental.

[jdunkerley79 · Accepted solution]

You are basically there - you've done the hard part the iteration.

The problem is the download tool. It is URL encoding the next page. Untick the box on the download tool of the Iterative Macro and it will work

[MarkEugene9000]

Thank you, that worked.

[ashkhan]

I seem to be stuck on a similar problem, i am trying to capture all the URL's for the next page in a table. In the main process that would be calling the macro i am expecting it to return a single row that gets added to build the table until it has reached it iteration but it only returns the last row from the iteration macro. Your insights will be very useful. 

Main wf that is calling the Iteration Macro above

[MarkEugene9000]

Hi Ashkhan,

In my example I was cycling through the pages, but I didn't record a master table of the page URLs like it appears you are trying to do. It seems like your table is overwriting the URl row each time it retrieves one, rather than listing them all together.

My immediate thought is, have you tried incorporating a union tool between the end of the MACRO with your table tool?

Mark

[ashkhan]

I tried it a couple of ways but the iterative macro is only returning one row, and i was expecting it to return one row per iteration that gets written to a table.

[jdunkerley79]

Could you post your workflow and macro so we can take a look. Happy for you to DM me if you want.

[ashkhan]

I have finally figured out what my problem was i had one output in the macro that was also used for iterating, i needed to add an additional output that would be the result of every iteration, that seemed to have worked nicely. 

.

[bb213]

Could use some help with this one. The api puts a limit of 1000 records at a time.  I need to iterate x number of times to get all the records. Having a hard time getting the url to update. 

The URL looks like this: https://api.pushshift.io/reddit/search/submission/?subreddit=nova&sort=desc&size=14500

I strip off the number at the end and calculate the number of iterations that I need to do.  Then I want to run the workflow that many times, bucketing the results along the way, and recreating the URL so that size decrements each time. Needless to say, it doesn't work. 

[john_miller9]

@bb213,

Please start a new post to get help with your challenge.  Only the original author of a post can accept a response as the correct solution, and this post already has an accepted solution.  You are much more likely to get a response that way.