Pagination (EndCursor) Iteration Macro

Question

Hi All,

With the help of this community i have attempted to build my first literation macro.

Overview:

* I have a workflow which i reference a Query (where i download the data and json parse the results)
* I have been successful at obtaining a single JSON string that I then parse into a table
* The query needs to dynamically change based on a parameter (endcursor) within the query

Problem Statement

Using the examples online, the input of my macro is a "text input" and using the "Iteration Query" . I then use the formula to download the Data and then use a Json Parse. 
Further along the chain i can see i am pulling out the "EndCursor" To Create a new Query.

When i run the macro, i am returning data, however its the Data from the "Text Input" and not the data from the Database
 
Query

query{
  getAllCompanies(companyLevel : "Company", pageControl:{first:2}){
  totalCount
    pageInfo {
      endCursor
      hasNextPage
      startCursor
      hasPreviousPage
    }
    edges {
      cursor
      node {
        companyId  
        name
        }
      }

Query Results

{
  "data": {
    "getAllCompanies": {
      "totalCount": 20000,
      "pageInfo": {
        "endCursor": "Q0RNQzAwNDF4NE90ak54WUM0bDE=",
        "hasNextPage": true,
        "startCursor": "Q0RNQzAwM2lBdThjYWc4UGR3cjU=",
        "hasPreviousPage": false
      },
      "edges": [
        {
          "cursor": "Q0RNQzAwM2lBdThjYWc4UGR3cjU=",
          "node": {
            "companyId": "CDMC003iAu8cag8Pdwr5",
            "name": "ABC Testng"
          }
        },
        {
          "cursor": "Q0RNQzAwNDF4NE90ak54WUM0bDE=",
          "node": {
            "companyId": "CDMC0041x4OtjNxYC4l1",
            "name": "Test 1"
          }
        }
      ]
    }
  }
}

Iteration Query

query{
  getAllCompanies(companyLevel : "Company", pageControl:{first:2,after:"Q0RNQzAwNDF4NE90ak54WUM0bDE="}){
  totalCount
    pageInfo {
      endCursor
      hasNextPage
      startCursor
      hasPreviousPage
    }
    edges {
      cursor
      node {
        companyId
        source
        name
          }
        }
      }
    }

Unfortunately i am unable to upload the workflow, but i have attached a screen shot, and the iteration macro

Looking forward to your assistance

Regards
Masond3

testing Iteration1a.yxmc

Macro.PNG

Workflow.PNG

SPetrie · Answer

You need to pass all those items through from the main workflow and into your macro input so the download tool can use it.

After it goes through the download tool, the same number of fields need to be sent back to iterate.

If there are 4 fields going into your macro, your iteration output needs to be sending back 4 fields as well.

Based on the pictures, your macro input should have ENV, Authorization, RadixURL, and GraphQL columns which all get populated by the main workflow and should be mirrored in your macro input.

Keep your download tool setup to use the fields as you do now, but inside the macro instead. Your GraphQL field will get updated with the new endpoint as you have it, but the other 3 fields need to come along with it to the iteration output so they are available to the download tool on the next pass.

I attempted to update the macro with what I saw in your screenshots. I can only take it so far since I cant actually run the download tool but hopefully this gets you close. Sorry again for all the back and forth. We will get you there :)

testing Iteration1a.yxmc

Masond3 · Answer

@SPetrie  Thanks for the quick response.

So i have tried that in the macro, however i was getting various errors.

In the workflow, before the second download tool, i am getting various tokens to ensure i have access to the  DB. The second downnload tool this, information is still being passed

When i add this to the macro , the first warning is that the URL is missing ( i assume this is because its not in text input ( as i only have one column)

I have attached the download Screenshot from the workflow

Third Formula before  second download tool in workflow

Second Download Tool after third formula in workflow

SPetrie · Answer

Sorry, I think I may have caused a bit of confusion when I helped with the iterative macro previously.

Your download tool should be inside your iterative macro.

The formula tool with the output was only in the macro as a place holder since we cant actually use your download tool in testing.